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BACKGROUND OF THE INVENTION 

1. Related applications. 

5 [0001] The present invention claims the benefit of U.S. Provisional Patent Application 
Serial No. 60/461,307 filed on April 8, 2003 entitled Home Automation Control 
Architecture. 

2. Field of the Invention. 

[0002] The present invention relates, in general, to automation and control systems, and, 
1 0 more particularly, to software, systems and methods for implementing a user interface 
for home automation systems that adapts to a variety of controlled devices of various 
manufacturers. 

3. Relevant Background. 

[0003] Home automation systems enable control of lighting, heating and air 
15 conditioning, window shades or curtains, pool heaters and filtration systems, lawn 
sprinklers, ornamental fountains, audio/visual equipment, and other appliances. Home 
automation systems include relatively simple systems that control one or a few functions 
in a home to more elaborate systems that control multiple, disparate features. Home 
automation systems may be integrated with a home security system so that when a fire 
2 0 alarm is raised, for example, internal and external lights will be turned on. 
Entertainment equipment such as audio, video, and home theatre equipment are 
available with control interfaces that enable a remote device to activate the equipment 
according to programmed schedules or remotely input commands. 
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[0004] In general, a home automation or control system comprises one or more 
controlled devices, one or more controllers, and a command communication link 
coupling a controller to a controlled device. The controllers may be directly 
programmable in which case they include some form of human interface for setting 
5 switches, event timing, and the like. Alternatively, controllers may be indirectly or 
remotely programmable in which case a separate human interface may be implemented 
by a personal computer or the like. Systems may be programmed using either a simple 
command language or using a graphical user interface that requires a computer with a 
monitor. These systems are expensive and require substantial investment by the user in 
1 0 time and energy to install and modify programming. To enter and/or change a program, 
a user must consult a user's manual or call a programming specialist. Hence, these 
systems are difficult to install and adapt to changing needs. Moreover, they are difficult 
to expand by adding new controlled devices or new software to add functionality. 

[0005] The home automation market has been fractured because most of the automation 
15 control manufacturers address narrow, vertical market segments, and use proprietary 
interfaces to protect their market. For example, some leading control manufacturers 
offer systems that focus on heating, ventilation, and air conditioning (HVAC) systems 
control. These manufacturers have little interest in controlling lighting, security 
systems, entertainment systems, and the like as these markets are entirely foreign to 
2 0 them. Other manufacturers make, for example, home entertainment controllers that 
integrate various video and audio components, but the primary focus has been to offer 
integrated control over only their own components. As a result, consumers face an array 
of control systems that do not intemperate, and that have proprietary interfaces that are 
difficult to understand and program. Moreover, the interfaces are inconsistent with each 

2 5 other in the manner in which controls are accessed, displayed and operate so that a user 

must learn the unique interface features of each control system. Hence, as more 
controlled systems are added, the complexity for the user increases significantly as new 
control interfaces must be added and learned. 

[0006] Some efforts have been made to provide integrated interfaces— single devices 

3 0 that "talk" to various control systems in a residence. One available system offers a rigid 
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architecture that is easy to install because it offers few customization options, however, 
the rigid architecture limits its functionality. Other systems offer more flexible 
interfaces, but in each case the implementations include limitations that make the 
products expensive and/or difficult to install. Further, the greater flexibility has a 
5 tendency to make the interfaces more difficult to use. A typical interface may be a 
numeric keypad, for instance, that requires the user to learn and remember what each of 
the keys or sequence of keys controls. Alternatively, users will attach labels to the keys 
to indicate what is controlled, which a useful but unsightly expedient. Accordingly, a 
need exists for a control system architecture that supports both flexible, but easy to learn 
1 0 and operate user interfaces. 

[0007] Server-based control systems involve a central control mechanism or server that 
issues commands to each of the controlled devices either directly, or through 
subordinate controllers. Server-based systems may be easier to program as the operator 
may need to become familiar with a single program, but are more complex to install as 

15 each of the controlled devices must be coupled to and in communication with the central 
server. The central server may implement a graphical programming environment to ease 
programming, however, user interfaces that enable users to operate controlled devices 
tend to remain non-graphical (e.g., switches and keypads). Moreover, because the server 
must be programmed to interact with the various controlled devices and/or subordinate 

2 0 controls, the operator must still become intimately familiar with the protocols and 
vagaries of each controlled device, defeating the advantages of a single software 
interface. 

[0008] Another common limitation of control systems arises from the control interface 
of the controlled devices themselves. A typical controlled device will implement a 

2 5 single control interface for receiving commands from a controller. This single interface 

is usually restricted to a single signaling protocol that makes a subset of the controlled 
devices functions accessible to the controller. Hence, the controlled device is designed 
to interact with a single controller and is unable to interact with a plurality of controllers. 
Further, the functionality that can be implemented is restricted by the controller 

3 0 hardware and/or software and cannot be readily extended. 
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[0009] Hence, a need exists for a home automation and control architecture that is easy 
to install, easy to use, and at the same time flexible and extensible to accommodate new 
devices and new functionality. 



SUMMARY OF THE INVENTION 

5 [0010] Briefly stated, the present invention involves a home automation and control 
architecture that includes one or more controlled devices and one or more control panels 
coupled by a communication network. Each of the controlled devices is directly coupled 
to at least one control panel and configured to communicate control information with the 
coupled panel. An inter-control communication path couples each of the control panels 
10 such that any one of the control panels can effect control over any of the controlled 
devices by communicating through another control panel that is directly coupled to a 
specific device. 

[0011] In another aspect, the present invention implements discovery processes in at 
least one of the control panels where the discovery processes interrogate other control 
15 systems and subsystems. Specifically, the discovery processes operate to interrogate 
controlled devices and subsystems to learn device-specific signaling protocols for 
communicating control information with the interrogated systems and subsystems. User 
input is translated to the learned signaling protocol appropriate for a particular control 
system or subsystem to which the user input is directed. 

2 0 [0012] In another aspect, the present invention comprises an automation system having 
an "open architecture" that supports standard communication protocols to transport 
communication with a variety of networked devices irrespective of the particular 
command interface and instruction/command architecture implemented by that device. 
An automation system will include a plurality of subsystems where at least some of the 

2 5 subsystems have disparate command interfaces. One or more controllers implement 
interfaces for communicating with the disparate command interfaces. The one or more 
controllers also implement a common user interface. 
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[0013] In yet another implementation the present invention involves an automation and 
control system which includes a message broker component. A plurality of network 
connected nodes implement interfaces for handling command communication, including 
control messages and status messages, with controlled devices. At least one message 
5 broker is coupled to receive command messages and direct received command messages 
to a selected node, wherein the node is selected based on that node's ability to handle the 
command message. In this manner, the message broker enables, inter-node 
communication that hide the physical installation details from a node user. In another 
implementation, the message broker selects nodes based on current context information 
1 0 related to the nodes such that commands can be routed based upon the context of a node 
sending the control message and/or context of a node selected to implement the 
command message. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0014] Fig. 1 shows an networked control environment in which the present invention is 
1 5 implemented; 

[0015] Fig. 2 illustrates a hardware-oriented view of a control panel device in 
accordance with the present invention; 

[0016] Fig. 3 illustrates a logical view of processes implemented by a control panel 
device in accordance with the present invention; 

2 0 [0017] Fig. 4A-4E show exemplary user interface views illustrating operation of the 
present invention; and 

[0018] Fig. 5 illustrates in story-board format an exemplary screen flow illustrating 
transitions of a user interface in accordance with the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

2 5 [0019] In general the present invention relates to a control architecture that distributes 
control information such as device commands and status information between human 
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interface units, referred to as control panels, and controlled devices. The device 
commands typically relate to operational commands to turn on/off a controlled device, 
adjust settings on a controlled device, query the status of a controlled device, and the 
like. Status information typically relates to current operational state of a controlled 
5 device, maintenance status of the controlled device, and the like. Moreover, the 
architecture of the present invention enables the human interface units to access external 
information from virtually any source supporting standard network communications 
including local area network resources, wide area network resources, Internet resources 
and the like. These types of resources have, until now, been unavailable to automation 
1 0 control systems. 

[0020] The present invention is particularly useful in home automation environments 
because it builds on top of the vast array of controlled devices and subsystems that 
already exist for managing lighting, security systems, heating and air conditioning, 
window shades or curtains, pool heaters and filtration systems, lawn sprinklers, 
15 ornamental fountains, audio/visual equipment, and other appliances. Hence, while it is 
contemplated that the present invention may be adapted to handle special-purpose and 
proprietary controlled devices and subsystems, a particular advantage is that the present 
invention adapts to existing controlled devices and subsystems and leverages their 
advantages. 

2 0 [0021] In essence, the present invention provides a control system architecture that 
abstracts the human interface organization from the physical interconnection to 
controlled devices. This enables a configurable many-to-many relationship to exist 
between controllers and controlled devices, greatly easing installation of a home 
automation system. In an exemplary installation, the controlled environment is defined 

2 5 by a series of linked screens so that each screen can include design elements, buttons 
and controls that represent the controlled environment in a contextually relevant fashion. 
This architecture is rigid enough to provide a consistent, predictable installation, but 
flexible enough to allow easy expansion and accurate representation of the environment, 
even dynamically changing environments. 
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[0022] The present invention is illustrated and described in terms of a distributed 
computing environment having nodes distributed throughout a building. However, an 
important feature of the present invention is that it is readily scaled upwardly and 
downwardly to meet the needs of a particular application. Accordingly, unless specified 
to the contrary the present invention is applicable to significantly larger, more complex 
network environments such as wide area networks (WANs), as well as small network 
environments such as conventional local area networks (LAN) systems or non- 
networked environments. 

[0023] Fig. 1 shows an exemplary control environment in which the present invention is 
implemented. A plurality of control panels 101 implement a programmable human 
interface in the particular embodiment. Control panels 101 are conveniently 
implemented using computer industry standard components and software to the extent 
practical, although special purpose, non-standard components and software are a 
suitable equivalent in particular applications. In a particular example, control panels 101 
comprise variants of a personal computer (PC) architecture to take advantage of price 
and performance features of the personal computer market. Control panels 101 are 
mounted throughout a building at locations where it is convenient or desired to exercise 
control over controlled systems. For example, a control panel 101 can be provided in 
each bedroom of a house, as well as a kitchen, office, entertainment areas and the like. 
Alternatively, one or two control panels 101 may be provided in central locations for 
shared access by all members of a household. 

[0024] As a particular example, the Companion™ 6 and Companion™ touch-screen 
interface units produced by CorAccess Systems of Golden, Colorado, assignee of the 
present invention, are suitable implementations for control panels 101. These devices 

2 5 implement a touch-screen graphical user interface and are compact flat screen devices 

that are readily wall mounted. These devices have suitable computing power and 
resources to implement a variety of applications for exercising home automation and 
control functions contemplated by the present invention. 

[0025] A hub 103, such as a conventional Ethernet/network hub, provides a network 

3 0 interconnection between control panels 101 and other devices. Hub 103 may be 
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implemented as a hub, router, switch, access point, or similar device that couples 
network devices. While Ethernet transport is used in the particular implementations 
described herein, other standard and/or proprietary transport mechanisms such as RS- 
232, RS-485, IEEE 1394, IEEE 802.1 1, and the like are suitable substitutes. Moreover, 
5 while the particular examples use an IP protocol, other protocols such as NetBIOS, 
AppleTalk, and the like may be appropriate in particular installations. Hub 103 may 
implement any number of ports to meet the needs of a particular application, and may be 
implemented by a plurality of physical devices to provide more ports and/or a more 
complex network including sub-networks, zones, and the like. 

10 [0026] In addition or alternatively, the present invention may be implemented using 
wireless networking components such as a wireless access point/router 105 and wireless 
control panels 107. When used in combination with a wired network, access point 105 
may be coupled to the network via hub 103. Alternatively, access point/router 105 may 
implement the hub/router/switch functionality to replace hub 103 altogether. Wireless 

15 control panels 107 implement similar functionality to control panels 101 and may be 
implemented by devices such as a Mobile Companion™ or Mobile Companion™ X 
available from CorAccess Systems. Wireless control panels 107 may also be 
implemented by a variety of wireless general-purpose computing devices such as laptop 
computers, handheld computers, palm computers and the like as well as special purpose 

2 0 devices provided in the form of, for example, remote controls, key fobs, smart cards, and 
the like. It is also contemplated that a wireless control panel may be implemented 
without an integrated graphical display, and instead use a detached display such as a 
television to implement a graphical user interface. This implementation would allow a 
control panel to be quite small as would be convenient for a handheld device. 

2 5 [0027] Yet another type of client, a virtual client 107 1 , may implement relatively 

lightweight processes that essentially emulate a control panel interface, and other 
processes that couple to a control panel 101. This is desirable when using some 
handheld computing devices for the virtual client as the handheld device can receive 
user commands and display status messages, but more computationally intense tasks are 

3 0 implemented in a control panel 101. Moreover, the virtual client 107 may be 
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implemented in any computing device (e.g., a work personal computer, handheld 
computer) that is not connected to the home network to provide the user interface 
features without requiring installation of the entire system. 

[0028] Several basic types of controlled devices are shown in Fig, 1. First, controlled 
5 devices may be directly connected to the network via, for example, hub 103. For 
example, an IP camera 109 comprises a camera that implements its own IP interface. A 
variety of security, telecommunications, environmental sensors, and the like are 
available with suitable IP interfaces. These controlled devices communicate control 
messages with a network-coupled control panel 101 or 107. 

10 [0029] A second type of controlled device is coupled to a particular control panel 
101/107 through a subsystem interface. For example, one control panel 101 couples to a 
lighting control subsystem 113 while another control panel 101 couples to an 
entertainment control subsystem 115. The subsystem interfaces comprise, for example, 
a control device that is provided with a particular third-party subsystem that may have a 

15 special-purpose or proprietary signaling protocol. The control panel 101 couples to the 
subsystem interface using the physical, electrical, and signaling protocols adopted by 
that subsystem. For example, a serial connection such as an RS-232 or RS-485 
connection is used in many cases. 

[0030] Alternatively, a subsystem interface may couple with hub 103 such as the case 
2 0 with analog subsystem interface 117. A variety of controlled devices are available such 
as security cameras, landscape controllers, telephony devices, HVAC systems, and the 
like that do not communicate using standard computer protocols. An analog subsystem 
interface 117 implements control functions to the extent possible with such devices and 
provides a network interface for coupling to other systems. An example of such a 
2 5 system is a variety of XI 0 devices and controls marketed by XI 0 Wireless Technology, 
Inc. of Seattle, Washington. 

[0031] The present invention also contemplates implementing shared services (e.g., 
telephony, Internet access, and the like) and/or resources such as shared mass storage 
111 through an internet gateway 127. Mass storage 111 may be coupled via hub 103, or 
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may be directly coupled to one or more control panels 101/107. Internet gateway 127 
may couple to hub 107 or may be integrated with hub 103 when implemented as a router 
or access point. Internet gateway 127 may implement a hardware and/or software 
firewall or other access control mechanisms for increased security. 

5 [0032] Optionally, one or more conventional personal computers 129 may be coupled to 
the network as well via hub 103 and/or wireless access point 105. In addition to 
implementing function similar to control panels 101/107, a personal computer 129 may 
implement applications that are not installed on or readily executable by control panels 
101/107. Additionally, personal computer 129 may implement common applications 
1 0 and/or computationally intensive applications such as word processing, web browsing, 
database access, and the like using conventional software. 

[0033] In operation, the system shown in Fig. 1 enables controlled many-to-many access 
between each control panel 101/107 and any of the controlled devices or subsystems. In 
prior home automation systems an entertainment control system was essentially stand 
15 alone, and could be accessed through a dedicated human interface that enabled control 
over audio equipment 123 and video device 121. A separate, independent system with a 
separate human interface would be required to control lighting, or security systems, for 
example. 

[0034] In contrast, the present invention enables any control panel 101/107 (or PC 129) 
2 0 to send a control message relevant to any particular controlled device. Each control 
panel 101/107 is aware of controlled devices and subsystems that it can directly access, 
and implement a message broker process that listens for control messages relevant to 
those devices/subsystems. The message broker is able to generate command and control 
messages directed to any other control panel 101/107 as well as respond to command 
2 5 and control messages from other control panel 101/107. 

[0035] In this manner the message broker processes enable each control panel 101/107 
to act as a server, which has distinct advantages over centralized server systems. For 
example, the system can continue to function in the event of failure of one control panel 
or the link to a control panel. The server programs (e.g., the message broker and the 
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HTTP server) can support "client" processes within the control panel or from other 
control panels in a unified fashion. 

[0036] The message broker server processes interpret the command messages and 
generate appropriate device-specific signaling to implement a command on the 
5 controlled device or subsystem. Similarly, status signals from controlled devices are 
interpreted and the message broker process generates network-compatible control 
messages that are distributed to the control panels 101/107. In this manner, any control 
panel 101/107 can communicate control information with any controlled device or 
subsystem without requiring detailed knowledge of the particular interface and signaling 
1 0 requirements of that controlled device or subsystem. 

[0037] Another feature of the present invention involves system discovery processes 
implemented in control panels 101/107. When a control panel 101 is coupled to a 
controlled device or subsystem, it interrogates that device or subsystem to learn details 
of the control interface of that particular system. Many special purpose subsystems 

1 5 support such interrogation to various degrees, and such interrogation will often provide 
sufficient detail to enable full access to even proprietary control interfaces. This 
interrogation may simply be a matter of determining the controller type in which case 
the control panel 101 can look up a command set and signaling protocol information for 
that controller type. Alternatively, the interrogation may reveal more details about 

2 0 actual commands that are available. In some cases, a controlled device or subsystem 
will return insufficient information during interrogation in which case the control panel 
can be manually or semi-automatically programmed to support that controlled device or 
subsystem. 

[0038] Because control panels 101/107 speak a common language amongst themselves, 

2 5 once a single control panel 101/107 discovers a particular controlled device or 

subsystem, that information can be readily shared (when desired by the users) with any 
other control panel 101/107. In most cases it is not necessary for every control panel 
101/107 to have detailed knowledge of a particular controlled device or subsystem. 
Instead, it is sufficient to be aware of the existence of each controlled device and the 

3 0 functionality available from that device. For example, details of entertainment control 
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subsystem 115 are often not important so long as information about audio system 123 
and its functions (on/off, signal source, volume control, status, etc.) are made available. 

[0039] Fig. 2 illustrates a hardware-oriented view of a control panel 101 in accordance 
with the present invention. A control panel 101 is powered by AC or DC power source 
5 although power supply and distribution are not shown in Fig. 2 to ease understanding of 
the present invention. A processor 201 implements data processing functionality for 
accessing and manipulating data from various subsystems and memory. Memory such 
as random access memory 203 and/or read only memory 205 may be provided as 
separate devices or integrated with processor 201. Processor 201 may be implemented, 

1 0 for example, by a Pentium® class processor provided by Intel Corporation, or the like. 
By using general purpose, widely available processor components a wide variety of 
operating system and application software is available, and new applications are easily 
developed. Alternatively, some processor architectures such as the Super-H licensed by 
SuperH, Inc. and StrongArm processors provided by a variety of manufacturers support 

15 integration of various functions such as serial interfaces, network interfaces, graphics 
subsystems, audio subsystems, and the like which may provide cost and/or performance 
benefits in some applications. 

[0040] The various subsystems shown in Fig. 2, including serial interfaces, network 
interfaces, graphics subsystems, audio subsystems, and sensor I/O, are exemplary only 
2 0 as additional subsystems may be useful in some applications, whereas some of the 
illustrated subsystems may not be required. Processor 201 is coupled to various 
subsystems using any available connection technology such as a peripheral component 
interconnect (PCI) bus or the like. 

[0041] Control subsystem interface(s) comprise one or more interfaces that support 

2 5 connection and communication with subsystems 113 and 1 15 in Fig. 1 . These would be 

typically implemented as serial interfaces coupling to, for example, an RS-232, RS-485, 
and/or universal serial bus physical connections. Any number of such interfaces may be 
provided in a control panel to meet the needs of a particular application. A network 
interface implements the resources required to support packet communication over, for 

3 0 example, a CAT-5, EEEE-4 or USB connection, for example. These functions are 
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substantially similar to what might be found in a convention personal computer network 
interface card (NIC). 

[0042] A graphics subsystem preferably supports an LCD panel display and touch- 
screen functionality. Alternatively, other graphical user interface I/O technology can be 
5 substituted in particular applications. LCD panels provide low power, convenient 
displays with long life and form factors that are amenable to wall mounting, and so are 
desirable in many applications. In applications where a GUI is not desired, a graphics 
subsystem can be greatly simplified by substituting driver electronics for LED and push- 
button human interface components. 

10 [0043] Optional audio subsystem may be provided to drive integrated speakers. 
Similarly, some sensory I/O may be desired to sense room temperature, or motion 
detectors to sense activity in proximity with a control panel 101. Sensory I/O may be 
omitted where desired, or provided through a serial connection, or provided through the 
network in a manner similar to other controlled devices and subsystems. 

15 [0044] Fig. 3 illustrates a logical view of processes implemented by a control panel 101 
device in accordance with the present invention. The drivers layer interfaces with the 
various hardware components shown in Fig. 2. Drivers may be added and removed 
from the drivers layer to support additional or updated functionality. Operating system 
layer may be provided by any available operating system, although it is useful to have an 

2 0 operating system that has a relatively small resource consumption such as Linux, 
Windows CE ® or the like. 

[0045] The application programming interface (API) layer comprises various processes 
that provide access to OS services and augment OS services for use by particular 
applications. Universal Plug-and-Play (UPnP™) processes support common protocols 
2 5 and procedures intended to enhance interoperability among network-enabled PCs, 
appliances, and wireless devices. Flash processes implement services related to 
Macromedia® FLASH programming environment and extensions. The particular 
implementation also includes web server processes such as provided by Xitami® web 
server products. The web server processes support a browser-based graphical user 
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interface using reliable and scaleable software that is readily configured to access other 
processes and resources. Further, web server processes support software updates well. 

[0046] Platform drivers are similar to hardware drivers, but offer more complex and 
platform specific functionality for devices such as hardware switches, LED indicators, 
5 and an LCD display. In particular implementations, the API layer includes driver 
libraries for accessing and operating hardware functions that are somewhat unique to a 
particular control panel 101/107. By making these drivers accessible to application 
plug-ins, the tasks involved in developing new plug-ins are greatly simplified. It 
becomes unnecessary to have intimate, detailed knowledge of how to turn on an LED or 
1 0 detect a switch activation, for example, because the built-in driver library can handle the 
details of these tasks. 

[0047] An important feature of the API layer is a message broker that provides services 
that coordinate communication between the various other API layer and application 
layer processes. The message broker component includes processes for listening to 

1 5 control messages, including command and status messages, and parsing those messages 
to determine which processes, if any, in that control panel 101/107 are involved in 
handling the control message. For example, a control message that is received 
externally and indicates a command to sound an alarm at that control panel 101 will be 
passed to an audio and/or security application plug in. A command message relating to 

2 0 turning on/off a light fixture may be passed to a home control plug in, or may be ignored 
if it relates to a fixture that is not coupled to that control panel 101. Similarly, a status 
message indicating that a light fixture is turned on may be formed into a message 
directed to one or more other control panels 101/107 that have interest in the status of 
that light fixture. The message broker can use, in particular embodiments, available 

2 5 mail protocols and the like to send notifications to external systems or recipients as well 
as communicating in-network messages to other control panels 101/107. These 
notifications can be used to convey information about events (e.g., a security alarm 
trigger), as well as system status (e.g., a communication failure with a control panel 
101/107 or failure of a HVAC subsystem). 
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[0048] Sophisticated configurable functionality is implemented by application layer and 
plug- in components. In a particular configuration, control panel 101 implements a 
native GUI that interfaces with various special-purpose plug-in components. Each 
control panel 101 may have a distinct set of plug-in components to meet the needs of the 
5 particular room or environment where the control panel 101 is installed. Further, the set 
of plug-in components can be changed based on specific user preferences. For example, 
a children's room may not normally use access to a home security system, however, 
when that same room is used as a guest room security system access may be enabled by 
installing a security plug-in. Similarly, the functionality of each component can be 
1 0 altered to increase or decrease the functionality based on the current user of the control 
panel, or the time of day, or other configurable permissions basis. 

[0049] Exemplary plug-in components include a home control component that is 
designed to interface with one or more controlled devices or subsystems. An intercom 
plug-in provides intra-building communications and/or interface to a telephony system. 

15 A photo plug-in handles accessing and displaying photographs, video, or other content 
while an audio plug-in enables controls that can play audio files on the control panel 101 
and/or control audio equipment 123 shown in Fig. 1. Security plug-in monitors status of 
a home security system and may enable features of the security system to be 
enabled/disabled under user control or programmatically. Any number of third party 

2 0 plug-ins are possible to implement extended functionality and/or enable access to new 
types of controlled devices and subsystems. Thirci party plug-ins are designed to comply 
with the API layer, or in some cases may interact with the operating system directly as 
suggested by the component labeled "OTHER" in Fig. 3. 

[0050] Various plug-in components are enabled to communicate with each other 
2 5 through the message broker component as well as communicating with other control 
panels 101/107 and controlled devices and subsystems. For example, a security plug-in 
may monitor status of a home security system and when an anomaly is detected, activate 
the audio and home control plug-ins to provide information and/or alerts to users. 
Moreover, the security plug-in may override and close certain applications such as a 
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photo player plug-in or audio plug-in to disable activities that might distract from the 
security plug-in's activities. 

[0051] Optionally, a browser user interface is provided to supplement the native GUI 
interface. So much software and plug-in functionality is available for common browser 
5 GUIs such as Mozilla, Internet Explorer, and the like that it may be useful to provide a 
browser interface and web plug ins to perform certain functions that augment and/or 
replace various other plug-ins. It is contemplated that a system may be provided that 
will eliminate the native GUI and plug-ins entirely and use only a browser interface with 
a suitable plug in to the API via web server processes in the API. 

1 0 [0052] Fig. 4A through Fig. 4E (F) illustrate a series of screen displays implemented in 
a particular embodiment of the invention. It should be understood that the variety and 
composition of the screens, as well as the flow from screen-to-screen, is readily adapted 
to meet the needs of a particular application. Each screen includes a number of 
graphical and design elements that may be static or animated (i.e., change over time). 

15 Some of the elements indicate status of the system or of a controlled device or 
subsystem. Others of the elements are controls or buttons that generate messages to 
other control panels 101, generate messages to controlled devices or subsystems, or 
initiate a transition to a new screen. In the particular implementation, a common design 
theme is presented across the various screens shown in Fig. 4A through Fig. 4F so that 

2 0 the user is presented with a familiar, consistent environment across the various 
functions. At the same time, the functionality of certain controls will change from 
screen-to-screen and screen-specific controls are provided so that each screen reflects a 
contextually relevant view of the control functions. The various controls can be selected 
and operated using a pointing device or by touch screen inputs. 

2 5 [0053] Fig. 4A shows a "home" screen that would be, for example, the normal state of a 

control panel 101 from which other control functionality can be reached. The exemplary 
home screen includes a thermostat display indicating room temperature and/or outside 
temperature, either of which may be measured by the control panel 101 itself, or be 
obtained from a remote device or other control panel 101. The home screen also 

3 0 includes a display of the security system status, which in Fig. 4A is not armed. The 
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security system status includes various control buttons labeled "day", "night", "away" 
and "vacation" that are used to transition to other screens used to program and operate 
the security system. 

[0054] Common user interface elements include a series of buttons on the right-hand 
5 side of the display that initiate a transition to other screens. For example, the upper 
button iconically indicates a "control center" and when operated will cause a transition 
to a control center screen shown in Fig. 4B. A thermometer icon identifies a button that 
initiates a transition to a HVAC control screen. A lamp icon identifies a button that 
initiates a transition to a lighting control screen, while a speaker icon identifies a button 

1 0 that initiates a transition to the media control screen. The choice and selection of the 
buttons to be displayed on the home screen is readily adapted to a particular application, 
and those shown in Fig. 4A are provided for illustration only. Desirably, each screen 
includes some navigational buttons such as the "back" button in Fig. 4A. Also, each 
screen may provide a "tools" button that initiates a transition to a screen used to 

15 configure and manage the system and/or control panel 101 itself (e.g., adjust contrast, 
update software, and the like). 

[0055] Upon activating the control center button in Fig. 4A, the control center screen 
shown in Fig. 4B is presented. As can be seen by comparison of Fig. 4A and Fig. 4B, 
there is a common design theme between the screens, although each screen presents a 
2 0 contextually relevant set of controls. For example, the control center screen no longer 
needs to display a control center button, and so that button is "replaced" with a button 
bearing a "home" icon which, when activated, initiates a transition back to the home 
screen shown in Fig. 4A. The control center screen provides controls and display 
graphics showing a different set of detail and enabling a different set of functionality 

2 5 than the home screen shown in Fig. 4A. 

[0056] Upon activating the "away" button in either Fig. 4A or Fig. 4B a series of screens 
related to activation of a security alarm is initiated. In Fig. 4C, an "enter code" screen is 
presented to prompt the user to enter a security code in addition to a number of 
"standard" controls located in the right hand side of the screen. ' Again, the set of 

3 0 standard controls presents a consistent design with respect to placement and graphics, 
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but some of the control functions may change to present controls that are more relevant 
to the context of entering a pass code to arm a security system. Fig. 4C illustrates the 
great flexibility enabled by the graphical display of the present invention in that a 
familiar numeric keypad is implemented for entering a security code. The numeric 
keypad is not useful for many other functions such as turning on lights or playing music, 
but it is contextually relevant to the task of arming a security system. Similarly, 
alphanumeric or symbolic keypads may also be presented as desired. When a proper 
code is entered, the entry may be validated against a stored code by the security 
subsystem, or may be validated using processes within the control panel 101 itself. In 
this manner, the present invention can both extend existing security features as well as 
implement security features that do not exist in the underlying controlled system. 

[0057] Upon entry of a valid code, the system transitions to the "arming" screen shown 
in Fig. 4D. Again, the arming screen includes very contextually relevant information 
including graphical elements that clearly communicate that the alarm system is arming. 
Because the control panel 101 can present programmable, animated graphics, it becomes 
possible to present information, like an alarm countdown, using graphical techniques not 
possible in prior systems. Fig. 4D illustrates that in many automation tasks, the user 
interface requirements for entering information may vary significantly from the user 
interface requirements for displaying status information. Prior systems were forced to 
compromise to make a single interface that served both functions. However, the present 
invention allows the screen to retrain common, contextually-relevant features while 
altering components as needed to support both entering and displaying information. 
Upon activation of the alarm, the system transitions back to the home screen in the 
particular example. However, the home screen now appears as shown in Fig. 4D with 
updated information concerning the alarm status. Additionally, some elements may 
change color, size, or shape to indicate the new status graphically. For example, the 
disarm system graphic appears red in the particular example whereas the "arm system" 
graphic appears green in Fig. 4A. 

[0058] Fig. 4F illustrates another screen that relates to thermostat and HVAC 
scheduling, a common home and office automation task. In the particular 
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implementation, the screen shown in Fig. 4F is reached by touching the thermometer 
graphic on any screen, but may be linked to by other paths as well. Unlike merely 
setting a thermostat, HVAC scheduling is a somewhat complex task as it involves 
numerous set points that may change over time for both cooling and heating systems. 
5 The familiar programmable thermostat in many homes allows a user to define time 
spans during a day or week, then to apply thermostat settings to each time span. Other 
systems use alphanumeric keyboard entry to define time spans and thermostat settings. 
These devices do not enable a user to visualize the settings over a span of time (e.g., a 
day) which makes the task more difficult. Moreover, keyboard entry of program settings 
10 is laborious and difficult to adjust as desired. 

[0059] In the screen of Fig. 4F, the scheduling task is benefited by both the graphical 
display of information, and the ability for users to manipulate and enter data using the 
graphical screen. For example, spans of time can be defined by touching and dragging 
vertical indicators 401. Temperature set points can be established in any of the zones by 
1 5 dragging the color bars to a desired level within the zone. Prior automation system user 
interfaces can not or do not enable users to interact in this graphical manner to actually 
enter scheduling information. A similar interface can be used with lighting, sound 
volume, and other controlled devices that the user desires to vary over time according to 
a schedule. 

2 0 [0060] Fig. 5 illustrates an exemplary screen flow demonstrating how the present 
invention enables transitions from one screen to another screen in response to either user 
input or system events. A typical system may involve tens or hundreds of linked 
screens. Beginning with the home screen, each of the control center, monitor, lighting 
and media button initiate a transition to a particular screen. The user returns to the home 

2 5 screen by activating the "home" control on any given screen, or by system events such as 
a time out that cause a screen to automatically transition back to the home screen. The 
tools button initiates a transition to a tools screen that presents various tools for 
calibration, setting preferences, and the like. It should be apparent that not all of the 
screen-to-screen links are shown in Fig. 5 to ease illustration and understanding. For 
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example, in any given screen a control labeled "media", which is presented in many 
screens, would link to the media control screen shown in the lower corner of Fig. 5. 

[0061] Each screen shown in Fig. 5 includes common design elements as well as 
screen-specific or context-specific portions 501-508. Each of these context-specific 
5 areas include controls that display context-relevant information and/or enable a user to 
select context-relevant functions which will in turn initiate any number of other screens. 
For example, context area 501 in the home screen includes a control that displays 
thermostat information and, when activated, launches an HVAC scheduling screen such 
as shown in Fig. 4F. Context area 501 also includes alarm system information 
1 0 indicating current status of the alarm system as well as control, which when activated, 
initiate a transition to the enter code screen as shown in Fig. 5. 

[0062] The context area 502 in the enter code screen includes controls as described in 
reference to Fig. 4C and the arming screen includes controls as described in reference to 
Fig. 4D. Upon completion of the arming processes, the system initiates a transition back 
15 to the home page where the context-specific controls and graphics are updated to 
indicate the new state of the alarm system. This transition to the home page is an 
example of an "unsolicited" transition (i.e., a transition that is initiated by a system event 
or status change rather than by an explicit user input). 

[0063] Referring now to the monitor screen, context specific area 505 includes a media 
2 0 player that displays input streaming from one or more monitor cameras such as IP 
camera 109. Exemplary controls that may be useful in this context include a control to 
switch cameras, move a camera, focus a camera, switch between tiled and non-tiled 
views of multiple input streams, record the camera view, and the like. The context 
specific area 506 of the lighting control screen may include controls for selecting various 

2 5 lights throughout a building, turning the selected lights on and off, dimming lights, and 

scheduling times for light-operations. The media screen includes a context specific area 
507 that may include, for example, controls for selecting various media sources (e.g., 
music. CD, radio, or an external source, DVD, tape, television, slide show, and the like). 
These individual device selections will, in many cases, launch further screens that are 

3 0 specific to the operation of the selected device. The media screen in Fig. 5 also includes 
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a control linking to a media library or other network attached resource for storing media 
files. As with the HVAC and lighting applications, the media screen may implement 
scheduling functions to record/play media files according to a schedule executed at one 
or more times in the future. 

5 [0064] Although the invention has been described and illustrated with a certain degree 
of particularity, it is understood that the present disclosure has been made only by way 
of example, and that numerous changes in the combination and arrangement of parts can 
be resorted to by those skilled in the art without departing from the spirit and scope of 
the invention, as hereinafter claimed. 
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